139 research outputs found
Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions
Quantification of the stationary points and the associated basins of
attraction of neural network loss surfaces is an important step towards a
better understanding of neural network loss surfaces at large. This work
proposes a novel method to visualise basins of attraction together with the
associated stationary points via gradient-based random sampling. The proposed
technique is used to perform an empirical study of the loss surfaces generated
by two different error metrics: quadratic loss and entropic loss. The empirical
observations confirm the theoretical hypothesis regarding the nature of neural
network attraction basins. Entropic loss is shown to exhibit stronger gradients
and fewer stationary points than quadratic loss, indicating that entropic loss
has a more searchable landscape. Quadratic loss is shown to be more resilient
to overfitting than entropic loss. Both losses are shown to exhibit local
minima, but the number of local minima is shown to decrease with an increase in
dimensionality. Thus, the proposed visualisation technique successfully
captures the local minima properties exhibited by the neural network loss
surfaces, and can be used for the purpose of fitness landscape analysis of
neural networks.Comment: Preprint submitted to the Neural Networks journa
Robotic Architectures
In the development of mobile robotic systems, a robotic architecture plays a crucial role in interconnecting all the sub-systems and controlling the system. The design of robotic architectures for mobile autonomous robots is a challenging and complex task. With a number of existing architectures and tools to choose from, a review of the existing robotic architecture is essential. This paper surveys the different paradigms in robotic architectures. A classification of the existing robotic architectures and comparison of different proposals attributes and properties have been carried out. The paper also provides a view on the current state of designing robot architectures. It also proposes a conceptual model of a generalised robotic architecture for mobile autonomous robots.Defence Science Journal, 2010, 60(1), pp.15-22, DOI:http://dx.doi.org/10.14429/dsj.60.9
Particle swarm optimization with crossover : a review and empirical analysis
Since its inception in 1995, many improvements to the original particle swarm
optimization (PSO) algorithm have been developed. This paper reviews one class of such
PSO variations, i.e. PSO algorithms that make use of crossover operators. The review is
supplemented with a more extensive sensitivity analysis of the crossover PSO algorithms
than provided in the original publications. Two adaptations of a parent-centric crossover
PSO algorithm are provided, resulting in improvements with respect to solution accuracy
compared to the original parent-centric PSO algorithms. The paper then provides an extensive
empirical analysis on a large benchmark of minimization problems, with the objective to
identify those crossover PSO algorithms that perform best with respect to accuracy, success
rate, and efficiency.http://link.springer.com/journal/104622017-02-20hb201
Empirical Loss Landscape Analysis of Neural Network Activation Functions
Activation functions play a significant role in neural network design by
enabling non-linearity. The choice of activation function was previously shown
to influence the properties of the resulting loss landscape. Understanding the
relationship between activation functions and loss landscape properties is
important for neural architecture and training algorithm design. This study
empirically investigates neural network loss landscapes associated with
hyperbolic tangent, rectified linear unit, and exponential linear unit
activation functions. Rectified linear unit is shown to yield the most convex
loss landscape, and exponential linear unit is shown to yield the least flat
loss landscape, and to exhibit superior generalisation performance. The
presence of wide and narrow valleys in the loss landscape is established for
all activation functions, and the narrow valleys are shown to correlate with
saturated neurons and implicitly regularised network configurations.Comment: Accepted for publication in Genetic and Evolutionary Computation
Conference Companion, July 15--19, 2023, Lisbon, Portuga
Seeking multiple solutions:an updated survey on niching methods and their applications
Multi-Modal Optimization (MMO) aiming to locate multiple optimal (or near-optimal) solutions in a single simulation run has practical relevance to problem solving across many fields. Population-based meta-heuristics have been shown particularly effective in solving MMO problems, if equipped with specificallydesigned diversity-preserving mechanisms, commonly known as niching methods. This paper provides an updated survey on niching methods. The paper first revisits the fundamental concepts about niching and its most representative schemes, then reviews the most recent development of niching methods, including novel and hybrid methods, performance measures, and benchmarks for their assessment. Furthermore, the paper surveys previous attempts at leveraging the capabilities of niching to facilitate various optimization tasks (e.g., multi-objective and dynamic optimization) and machine learning tasks (e.g., clustering, feature selection, and learning ensembles). A list of successful applications of niching methods to real-world problems is presented to demonstrate the capabilities of niching methods in providing solutions that are difficult for other optimization methods to offer. The significant practical value of niching methods is clearly exemplified through these applications. Finally, the paper poses challenges and research questions on niching that are yet to be appropriately addressed. Providing answers to these questions is crucial before we can bring more fruitful benefits of niching to real-world problem solving
Training multi-agent teams from zero knowledge with the competitive coevolutionary team-based particle swarm optimiser
A new competitive coevolutionary team-based particle
swarm optimiser (CCPSO(t)) algorithm is developed to
train multi-agent teams from zero knowledge. The CCPSO(t)
algorithm is applied to train a team of agents to play simple
soccer. The algorithm uses the charged particle swarm optimiser
in a competitive and cooperative coevolutionary training
environment to train neural network controllers for the
players. The CCPSO(t) algorithm makes use of the FIFA
league ranking relative fitness function to gather detailed
performance metrics from each game played. The training
performance and convergence behaviour of the particle swarm
is analysed. A hypothesis is presented that explains the lack
of convergence in the particle swarms. After applying a clustering
algorithm on the particle positions, a detailed visual
and quantitative analysis of the player strategies is presented.
The final results show that the CCPSO(t) algorithm is capable
of evolving complex gameplay strategies for a complex
non-deterministic game.http://link.springer.com/journal/5002017-02-28hb201
Set-based particle swarm optimization applied to the multidimensional knapsack problem
Particle swarm optimization algorithms have been successfully applied to discrete-
valued optimization problems. However, in many cases the algorithms have been tailored
specifically for the problem at hand. This paper proposes a generic set-based particle
swarm optimization algorithm for use on discrete-valued optimization problems that can
be formulated as set-based problems. A detailed sensitivity analysis of the parameters of
the algorithm is conducted. The performance of the proposed algorithm is then compared
against three other discrete particle swarm optimization algorithms from literature using the
multidimensional knapsack problem, and is shown to statistically outperform the existing
algorithms.http://www.springerlink.com/content/120597/?p=36e5205be3fa464a82382b977b16ece5&pi=2086hb201
Performance measures for dynamic multi-objective optimisation algorithms
When algorithms solve dynamic multi-objective optimisation problems (DMOOPs), performance
measures are required to quantify the performance of the algorithm and to compare
one algorithm’s performance against that of other algorithms. However, for dynamic multiobjective
optimisation (DMOO) there are no standard performance measures. This article
provides an overview of the performance measures that have been used so far. In addition,
issues with performance measures that are currently being used in the DMOO literature are
highlighted.http://www.elsevier.com/locate/insmv201
Benchmarks for dynamic multi-objective optimisation algorithms
Algorithms that solve Dynamic Multi-Objective Optimisation Problems (DMOOPs) should be tested on benchmark functions to determine whether the algorithm can overcome specific difficulties that can occur in real-world problems. However, for Dynamic Multi-Objective Optimisation (DMOO), no standard benchmark functions are used. A number of DMOOPs have been proposed in recent years. However, no comprehensive overview of DMOOPs exist in the literature. Therefore, choosing which benchmark functions to use is not a trivial task. This article seeks to address this gap in the DMOO literature by providing a comprehensive overview of proposed DMOOPs, and proposing characteristics that an ideal DMOO benchmark function suite should exhibit. In addition, DMOOPs are proposed for each characteristic. Shortcomings of current DMOOPs that do not address certain characteristics of an ideal benchmark suite are highlighted. These identified shortcomings are addressed by proposing new DMOO benchmark functions with complicated Pareto-Optimal Sets (POSs), and approaches to develop DMOOPs with either an isolated or deceptive Pareto-Optimal Front (POF). In addition, DMOO application areas and real-world DMOOPs are discussed.http://surveys.acm.orghj201
- …